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Introduction 

A  Software  Development  Notebook  (SDN)  technique 
is  described  in  this  paper.  Significant  benefits  have 
resulted  from  the  use  of  this  technique  over  the  past 
three  years.  Software  Is  developed  in  a  more  orderly 
and  disciplined  manner.  The  emerging  software  pro¬ 
duct  is  visible  and  auditable.  Deficiencies  are  iden¬ 
tified  and  corrected  earlier.  Documentation  evolves 
in  the  required  format.  Software  and  documentation 
are  more  consistent.  Current  schedule  status  is 
available.  The  net  results  are  improved  software 
engineering  discipline  and  an  improved  software 
product. 

A  general  description  of  the  SDN  technique  was 
first  presented  in  a  prior  paper.  (Ref.  1)  The  sub¬ 
sequent  interest  shown  by  both  software  engineering 
and  software  R&QA  professionals  motivated  us  to 
prepare  this  paper  devoted  to  the  SDN  and  SDN  audits. 

The  question  "What  is  an  SDN?"  is  answered  in  the 
next  section  of  this  paper.  This  is  followed  by  a  brief 
description  of  how  SDN's  are  established  and  main¬ 
tained.  SDN  audit  planning  and  conduct  are  then  ad¬ 
dressed.  In  conclusion,  lessons  learned  from  our  use 
of  the  SDN  technique  are  presented. 

What  is  an  SDN  ? 

To  the  programmer,  the  SDN  is  his  day-to-day 
working  notebook.  To  Computer  Software  R &QA,  the 
SDN  is  a  window  through  which  the  software  process 
and  the  emerging  software  product  are  viewed. 

A  Software  Development  Notebook  is  a  loose-leaf 
notebook  which  provides  a  common  collection  point  for 
all  current  information  relating  to  a  computer  program 
component  (CPC)*.  The  sections  of  the  SDN  are 
identified  in  Figure  1.  The  order  of  these  sections  is 
compatible  with  the  Software  Development  Process 
documented  in  reference  1. 

The  content  and  format  requirements  for  each  sec¬ 
tion  of  the  SDN  are  defined  in  detail  in  the  Software 
Standards  and  Procedures  Manual.  A  summary  of  each 
SDN  section  follows. 


*A  computer  program  is  partitioned  into  computer 
program  components  which  are  further  partitioned 
into  routines. 


Section  Q  -  Status  Sheets 

Section  0  of  the  SDN  contains  status  sheets  which 
document  schedule,  completion,  approval  and  audit 
status.  The  first  status  sheet  provides  an  overview  for 
the  CPC.  Detailed  status  sheets  follow  for  sections  2 
through  6  of  the  SDN.  These  status  sheets  are  useful  in 
combating  the  "90%  complete"  syndrome.  (Ref.  2) 

An  overview  status  sheet  is  shown  in  Figure  2. 

This  status  sheet  provides  visibility  of  the  development 
progress  and  audit  status  for  a  CPC.  Figure  3  shows  a 
typical  detailed  status  sheet  which  provides  similar 
visibility  to  the  routine  level. 

Section  1  -  Requirements 

Computer  Program  Performance  Specification 
(CPPS)  requirements  currently  allocated  to  this  CPC 
are  listed  in  Section  1.  The  listing  includes  CPPS 
paragraph  number,  requirement  number,  a  summary 
statement  of  the  requirement  and  the  names  of  the  rou¬ 
tines  which  implement  and  satisfy  the  requirement. 

Early  in  the  software  development  cycle,  some 
CPPS  requirements  will  be  missing  or  incomplete. 

Any  requirements  assumptions  made  so  that  software 
development  can  proceed  are  documented  in  this  sec¬ 
tion  of  the  SDN.  Copies  of  letters  and  memos  which 
relate  to  requirements  issues  are  also  included. 


This  section  serves  as  a  constant  reminder  of  cur¬ 
rent  requirements  as  the  CPC  is  designed  and  tested. 


Figure  1.  SDN  Sections 
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Section  2  -  Detailed  Design  Descriptions 


ments  to  CPC  and  routine  level  functional  capabilities 
Is  also  included. 


This  section  contains  the  current  detailed  design 
of  the  CPC.  It  typically  Includes  subsections  for  a  CPC 
overview,  a  data  base  description  and  the  description  of 
each  routine.  Significant  cost  savings  result  from 
tailoring  the  format  and  content  of  this  section  to  meet 
final  documentation  requirements. 

The  CPC  Overview  Subsection  provides  a  brief 
general  description  of  the  functions  performed  by  the  CPC 
and  includes  a  hierarchical  diagram  showing  its  structure. 
This  subsection  also  describes  all  external  CPC 
interfaces. 

The  CPC  Data  Base  Subsection  contains  a  complete 
description  of  the  local  variables  used  by  the  CPC  and 
identifies  the  routines  which  set  or  use  each  variable. 

The  format  and  content  of  tables  used  in  this  subsection 
and  data  base  naming  conventions  are  defined  in  detail 
in  the  Software  Standards  and  Procedures  Manual. 

The  Routine  Detailed  Design  Subsections  provide  a 
description  of  each  routine.  The  description  includes 
functions  performed,  enablement  criteria,  inputs,  out¬ 
puts,  a  processing  description  and  a  routine  data  base 
description. 

Section  3  -  Functional  Capabilities 

Functional  capabilities  of  the  software  design  are 
listed  in  this  section  and  traced  to  CPPS  requirements. 
They  provide  the  basis  for  developing  the  CPC  and 
routine  test  cases  described  later. 


Separate  functional  capability  lists  are  included  in 
this  section  of  the  SDN  for  the  CPC  and  for  each  rou¬ 
tine.  A  traceability  matrix  relating  CPPS  require¬ 
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Section  4  -  Code 

A  current  code  listing  for  each  routine  is  kept  in  a 
numbered  binder  at  a  central  location.  This  section  of 
the  SDN  contains  a  table  which  provides  a  cross  ref¬ 
erence  between  routines  and  binder  numbers. 

Section  5  -  Test  Case  Descriptions 

In  this  section  CPC  and  routine  level  test  cases  are 
described  and  traced  to  functional  capabilities  of  the 
software. 

The  description  of  each  test  case  includes  its  iden¬ 
tification  number  and  purpose,  inputs,  support  software 
requirements  and  criteria  for  successful  completion. 
Separate  traceability  matrices  are  also  included  for  the 
C  PC  and  each  of  its  routines  relating  functional  capa¬ 
bility  identification  numbers  to  test  case  identification 
numbers. 

Section  6  -  Test  Case  Results 

Like  code  listings,  hard  copy  results  of  tests  are 
kept  in  numbered  binders  at  a  central  location.  These 
results  are  clearly  marked  to  show  that  the  criteria 
for  successful  completion  have  been  met.  This  sec¬ 
tion  of  the  SDN  contains  a  table  which  provides  a  cross 
reference  between  test  case  numbers  and  binder 
numbers. 

Completion  and  approval  of  Section  6  indicate  that 
the  SDN  has  served  its  major  purpose  and  the  CPC  is 
ready  for  integration  testing.  Th“  SDN  will  continue 
to  be  used  for  software  modificatic..  and  change  control. 
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Figure  2.  Overview  Status  Sheet  Figure  3.  Detailed  Status  Sheet  j 
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Section  7  -  Software  Problem  Report  Log 

The  identification  and  reaolutlon  of  software  prob¬ 
lems  improve  the  reliability  and  quality  of  the  final 
software  product.  This  section  and  Section  8  of  the 
SDN  provide  visibility  and  control  of  these  activities. 

An  SPR  log  and  a  copy  of  all  Software  Problem 
Reports  (SPR)  affecting  this  CPC  are  contained  in 
Section  7.  SPR's  define  and  document  problems  iden¬ 
tified  during  integration  and  acceptance  testing.  They 
also  document  the  test  conditions  under  which  the  pro¬ 
blems  occurred.  Software  problem  reporting  and  the 
SPR  we  use  are  described  in  reference  1.  The  SPR  log 
we  use  is  shown  in  Figure  4. 

Section  8  -  Software  Change  Order  Log 

Section  8  contains  a  Software  Change  Order  (SCO) 
log  and  a  copy  of  all  SCO's  affecting  this  CPC.  SCO's 
document  and  control  changes  to  code.  SCO's  are  pre¬ 
pared  in  response  to  approved  CPPS  requirements 
changes  and  Software  Problem  Reports.  The  SCO  log 
we  use  is  shown  in  Figure  4. 

Section  9  -  Miscellaneous 

This  section  contains  material  relative  to  the 
design,  code  and  test  of  the  CPC  which  the  individual 
responsible  for  the  CPC  feels  is  appropriate  for  pre¬ 
sent  or  future  use.  This  material  typically  includes 
working  notes,  reference  tables,  technical  reports, 
memos  and  correspondence.  Copies  of  the  most  recent 
SDN  Audit  Report  and  responses  to  this  report  are  also 
included. 

A  general  description  of  the  Software  Development 
Notebook  has  been  provided.  It  is  important  that  the  de¬ 
tailed  format,  content  and  organization  of  the  SDN  for  a 
project  be  specified  in  the  Software  Standards  and  Pro¬ 
cedures  Manual  early  in  the  Software  Development  Pro¬ 
cess.  It  is  equally  important  that  this  manual  specify 
the  procedures  and  responsibilities  for  establishing  and 
maintaining  the  SDN. 

Establishing  and  Maintaining  the  SDN 

An  SDN  is  established  for  each  computer  program 


component  (CPC)  at  the  start  of  the  Detail  Design  Phase 
of  the  Software  Development  Process.  It  is  maintained 
current  throughout  the  remainder  of  the  development 
process. 

The  physical  establishment  of  an  SDN  is  a  clerical 
procedure  performed  by  the  project  librarian  under  the 
direction  of  the  Chief  Programmer.  It  entails  marking 
a  loose-leaf  notebook  with  the  project  and  CPC  name, 
inserting  labeled  tab  dividers  for  each  section  of  the 
SDN  and  placing  status  sheets  in  Section  0,  the  CPC 
requirements  list  in  Section  1  and  log  forms  in  Sections 
7  and  8.  The  SDN  is  then  provided  to  the  individual 
responsible  for  the  CPC. 

Based  on  a  completion  date  provided  by  the  Chief 
Programmer,  the  individual  responsible  for  the  CPC 
enters  due  dates  for  each  section  on  the  overview  status 
sheet.  Compatible  due  dates  for  each  routine  are  later 
entered  on  the  detailed  status  sheets  by  the  responsible 
programmer.  All  schedules  are  reviewed  and  approved 
by  the  Chief  Programmer.  As  a  subsection  or  section 
of  the  SDN  is  completed,  the  completion  date  is  entered 
on  the  appropriate  status  sheet.  After  completing  his 
review,  the  Chief  Programmer  initials  the  status  sheet 
to  indicate  approval.  Section  0  of  the  SDN  therefore 
provides  current  CPC  schedule  and  approval  status  at 
all  times. 

The  individual  responsible  for  the  CPC  augments 
the  requirements  list  in  Section  1  to  show  allocations 
to  the  routine  level  and  updates  the  list  as  these  alloca¬ 
tions  or  requirements  change.  He  also  enters  and  logs 
Software  Problem  Reports  in  Section  7  when  they  are 
issued  and  resolved,  and  enters  and  logs  Software  Change 
Orders  in  Section  8  when  they  are  approved  and  imple¬ 
mented.  The  inclusion  and  purging  of  material  in 
Section  9  is  also  the  responsibility  of  this  individual. 

The  remaining  sections  of  the  SDN  are  the  respon¬ 
sibility  of  the  computer  programmers.  They  describe 
the  software  design  and  design  capabilities  in  Sections 
2  and  3,  develop  the  code  and  test  cases  in  Sections  4 
and  5  and  document  test  results  in  Section  6.  Keeping 
the  SDN  current,  readable  and  compliant  with  the  soft¬ 
ware  standards  is  an  on-going  computer  programmer 
responsibility. 


Figure  4.  SPR  and  SCO  Logs 


3 


The  importance  of  maintaining  Software  Develop¬ 
ment  Notebooks  which  are  current  and  compliant  with 
software  standards  cannot  be  overemphasized.  Well 
maintained  SDN's  minimize  later  integration  problems 
by  providing  design  and  interface  visibility  to  other 
programmers.  A  costly  and  time  consuming  documen¬ 
tation  cycle  is  also  avoided  by  evolving  final  soft¬ 
ware  documentation  in  the  SDN.  The  procedures  de¬ 
scribed  above  have  been  used  on  several  major  pro¬ 
jects  over  the  last  three  years.  The  result  has  been 
improved  in-house  and  subcontractor  software  and 
documentation. 

SDN  Audits 

In  addition  to  their  value  as  a  software  develop¬ 
ment  technique,  SDN's  are  a  cost  effective  control 
and  assurance  technique.  They  provide  visibility  into 
the  Software  Development  Process  and  the  emerging 
software  product.  SDN's  facilitate  audit  activity  by 
providing  all  needed  information  in  one  location.  Our 
SDN  audits  are  responsive  to  the  requirements  of 
MIL-S-52779.  (Ref.  3) 

The  following  principles  guide  SDN  audit  planning: 

•  SDN  audits  must  be  an  element  of  a  compre¬ 
hensive  Software  Quality  Assurance  Program 
(Ref.  1)  which  should  be  documented  as  an 
integral  part  of  a  Software  Development  Plan. 

•  SDN  audits  should  be  conducted  by  individuals 
who  are  organizationally  independent  of  the 
software  developers. 

•  SDN  audits  should  begin  early  and  continue 
through  the  entire  development  process. 

•  SDN  audits  should  be  professionally  planned, 
conducted,  and  documented,  and  corrective 
action  should  be  followed. 

Our  planning  recognizes  the  need  to  shift  audit 
emphasis  for  different  software  development  activities. 
SDN  audits  and  audit  criteria  during  design,  code  and 
test  are  described  next. 

SDN  Audits  During  Design 

The  initial  SDN  audit  is  conducted  early  in  the 
Detail  Design  Phase  to  verify  that  SDN's  have  been 
established  for  all  identified  computer  program  com¬ 
ponents  (CPC's)  and  are  organized  in  accordance  with 
software  standards.  Sections  0  and  1  are  checked  at 
this  time  to  assure  that  a  schedule  and  a  requirements 
list  have  been  established. 

Later  audits  during  this  phase  confirm  that  Sec¬ 
tion  0  includes  detailed  schedules  and  that  completion 
and  approval  status  is  current.  The  requirements  list 
in  Section  1  is  checked  to  assure  that  it  is  up-to-date 
and  shows  requirement  allocations  to  the  routine  level. 

CPC  routine  and  data  base  design  descriptions  in 
Section  2  are  audited  for  compliance  with  software 
standards  and  for  consistency.  Audit  criteria  include 
format,  content  and  nomenclature  standards.  Flow¬ 
charts  or  fonctional  flows  are  checked  for  allowable 
constructs,  symbology  and  structure.  Data  tables  are 
checked  for  required  format  and  content.  Technical 
and  nomenclature  consistency  throughout  the  design 
description  are  emphasized  during  these  audits. 


available.  Computer  Software  R&QA  auditors  use 
detailed  status  sheets  in  Section  0  to  determine  code 
availability  and  the  table  in  Section  4  to  locate  code 
listings.  These  listings  are  checked  for  compliance 
with  coding  standards  such  as  structure,  annotation  and 
and  routine  size.  Agreement  between  the  code  listings 
and  the  approved  design  descriptions  is  also  verified. 
Instruction  sequences  and  data  usage  are  both  checked. 

SDN  Audits  During  Test 

SDN  audits  during  routine  and  CPC  testing  focus  on 
functional  capabilities  in  Section  3,  test  case  descrip¬ 
tions  in  Section  5  and  test  case  results  in  Section  6. 

The  auditor  first  checks  that  fonctional  capabilities 
have  been  identified  and  traced  to  requirements.  Test 
case  descriptions  and  associated  traceability  matrices 
are  then  audited  to  assure  that  all  functional  capa¬ 
bilities  will  be  tested.  Finally,  test  results  are  audi¬ 
ted  to  determine  that  all  tests  were  conducted  and 
satisfactorily  completed. 

The  emphasis  of  SDN  audits  during  integration  and 
acceptance  testing  is  to  assure  that  identified  problems 
are  resolved  and  approved  changes  are  implemented. 
These  audits  address  the  Software  Problem  Report 
(SPR)  logs  in  Section  7,  the  Software  Change  Order 
(SCO)  logs  in  Section  8  and  resulting  changes  in  other 
sections  of  the  SDN.  SPR  and  SCO  logs  are  checked 
against  configuration  control  board  records  to  verify 
that  the  logs  are  complete.  The  inclusion  of  all  logged 
SPR's  and  SCO's  is  then  checked.  These  documents 
are  used  to  verify  that  problems  have  been  resolved 
and  approved  changes  have  been  implemented. 

SDN  Audit  Planning,  Procedures  and  Documentation 

The  approved  Software  Development  Plan  includes 
a  section  describing  the  Software  Quality  Assurance 
Program.  SDN  audit  scope,  frequsney,  sample  selec¬ 
tion  and  documentation  are  defined  in  this  section.  An 
approved  implementation  plan  is  required  prior  to  each 
audit  which  defines  the  audit  objectives,  criteria, 
sample,  schedule  and  audit  team  personnel.  Audit 
objectives  and  audit  criteria  are  based  on  the  current 
software  development  phase,  the  results  of  prior 
audits  and  recommendations  from  Chief  Programmers 
and  managers.  The  audit  sample  is  selected  to  include 
work  from  each  computer  programmer.  Experience 
has  shown  that  routines  which  are  complex  or  signifi¬ 
cantly  behind  schedule  should  also  be  selected  for 
audit. 

Since  the  SDN  is  well  defined  and  self-contained, an 
Independent  audit  can  be  conducted  without  involving 
the  software  developers.  This  approach  results  in 
SDN  audits  which  are  objective  and  non-disrupMve.  A 
previously  prepared  checklist  is  completed  for  each 
SDN  audited.  The  auditor  identifies  the  subsection  of 
the  SDN  being  reviewed  and  then  describes  any  de¬ 
ficiencies.  The  audit  team  leader  uses  these  comple¬ 
ted  checklists  to  prepare  the  audit  report.  This  report 
documents  audit  findings  and  associated  recommenda¬ 
tions.  Prior  to  issuing  the  report,  the  audit  team 
leader  validates  the  technical  accuracy  of  audit  findings 
with  the  Chief  Programmer. 

Response  forms  are  provided  to  the  Chief  Program¬ 
mer  on  which  he  defines  corrective  actions  and  sched¬ 
ules  completion  dates.  Computer  Software  R&QA  uses 
these  completed  forms  to  verify  that  corrective  action 
is  defined  and  implemented. 


3DN  Audita  During  Code 

SDN  audit  emphasis  shifts  when  code  becomes 


Problems  have  been  Identified  by  SDN  audits  dur¬ 
ing  all  phases  of  the  Software  Development  Process. 
Cost  savings  were  realized  because  problems  identified 


early  in  the  process  are  far  less  difficult  and  less 
costly  to  correct.  (Ref.  4)  Moreover,  some  of  these 
problems,  if  not  found  and  corrected,  would  have  had 
major  impact  on  software  quality,  reliability  and 
maintainability. 

Experience  and  Results 

The  SDN  technique  described  in  this  paper  has  been 
applied  on  major  projects  over  the  past  three  years  to 
provide  control  and  assurance  for  both  in-house  and  sub¬ 
contractor  developed  software.  The  SDN  audit  tech¬ 
nique  has  led  to  the  acceptance  of  problem  identifica¬ 
tion  and  correction  activity  as  Integral  parts  of  the 
Software  Development  Process.  Significant  qualitative 
and  quantitative  results  from  this  experience  are  pre¬ 
sented  below. 

The  most  important  prerequisite  for  successful 
application  of  the  SDN  technique  is  the  Software  Stand¬ 
ards  and  Procedures  Manual.  This  manual  should  un¬ 
ambiguously  specify  the  details  of  SDN  format,  content 
and  organization.  The  SDN  should  be  easy  to  use  and 
maintain.  It  should  be  noted  that  improved  software 
standards  result  from  the  use  of  standards  as  audit 
criteria. 

The  support  of  functional  and  project  management 
is  needed  when  the  SDN  technique  is  Instituted.  Resis¬ 
tance  from  some  software  developers  should  be  anti¬ 
cipated.  The  technique  will  be  accepted  by  these  de¬ 
velopers  when  they  experience  its  benefits. 

We  have  found  that  a  Computer  Software  R&QA 
engineer  assigned  to  a  software  project  provides 
valuable  continuity  and  can  effectively  serve  as  the 
leader  of  the  independent  SDN  audit  team.  He  must  be 
able  to  read  code. 

Our  in-house  and  subcontractor  audit  findings  have 
led  us  to  some  expected  and  some  surprising  observa¬ 
tions.  Among  these  are  the  following: 


•  The  major  problem  identified  by  SDN  audits 
was  lack  of  consistency  between  design 
descriptions  and  code  listings. 

•  The  quality  of  a  programmer's  work  on  a 
project  remains  relatively  constant  and  varies 
widely  from  programmer  to  programmer. 

•  Frequent  requirements  changes,  schedule 
difficulty  and  personnel  turnover  can  adversely 
impact  software  quality. 


These  observations  should  be  considered  in  selecting 
audit  samples. 

Analysis  of  1175  deficiencies  identified  during 
monthly  SDN  code  audits  provided  the  quantitative 
results  which  follow.  Most  of  the  deficiencies  (842 
or  72%)  were  inconsistencies  between  code  listings  and 
design  descriptions.  An  example  of  this  type  of  de¬ 
ficiency  is  inconsistency  between  the  code  annotation 
and  the  design  description.  The  remaining  deficiencies 
(333  or  28%)  were  non-compliance  of  code  listings  with 
software  standards.  An  example  of  this  type  of  defi¬ 
ciency  is  a  routine  prologue  which  lacks  required 
content. 

As  Figure  5  illustrates,  commentary  deficiencies 
predominate  in  both  categories.  The  majority  of  these 
deficiencies  were  in  the.  documentation  rather  than  the 
code.  Correcting  them  improved  software 
maintainability. 

Summary 

The  Software  Development  Notebook  technique  is 
being  used  in  the  development  of  computer  software. 

The  success  of  the  technique  depends  upon  well- 
defined  software  standards  and  management  support. 

An  SDN  is  a  loose-leaf  notebook  which  provides  a 
common  collection  point  for  all  current  information 
relating  to  a  computer  program  component  (CPC).  It 
includes  schedule,  requirements,  design,  code,  test 
and  change  control  information.  The  establishment  of 
an  SDN  is  a  clerical  procedure.  It  is  maintained  by 
the  software  developers  who  must  keep  it  current. 

SDN's  are  a  cost  effective  control  and  assurance 
technique.  They  provide  visibility  and  facilitate  inde¬ 
pendent  audit  activity.  The  emphasis  of  audit  activity 
shifts  with  the  phases  of  the  Software  Development 
Process.  SDN  audits  must  be  professionally  planned, 
conducted,  documented  and  corrective  action  must  be 
followed. 

We  have  used  this  SDN  technique  over  the  past  three 
years  for  in-house  and  subcontractor  developed  soft¬ 
ware.  SDN's  are  easy  to  use  and  to  audit.  Our  audits 
are  led  by  an  independent  Computer  Software  R&QA 
engineer  assigned  to  the  project.  The  major  problem 
Identified  by  SDN  audits  was  lack  of  consistency  be¬ 
tween  design  descriptions  and  code  listings.  Improved 
software  maintainability  resulted  from  the  subsequent 
corrective  actions. 

The  "bottom  line"  is  an  improved  software 
product. 


Figure  5.  Code  Listing  Deficiencies 
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